Home | | Latest | About | Random
# 12 Matrix-vector product.
Great, now that we know how matrix multiplication works, we like to look at one specific situation $A\vec x$, when we have $A$ some $n \times k$ matrix, and $\vec x$ some $k \times 1$ matrix. Often, we refer to such $k\times 1$ matrix $\vec x$ as a **column vector**, or simply a **vector**. Note, however, later we will generalize the idea of vectors, so be prepared to be flexible later on.
Ok, let's say we have, as a starting example: $$
A = \begin{bmatrix}
4 & -3 & 2 \\
1 & 2 & 7
\end{bmatrix} \quad \text{and } \vec x = \begin{bmatrix}
x_{1} \\
x_{2} \\
x_{3}
\end{bmatrix}.
$$Note $A$ is $2\times 3$ and $\vec x$ is $3\times 1$, so the matrix-vector product $A\vec x$ is defined, and the result will be a $2\times 1$ column vector. Let us multiply it out: $$
A\vec x = \begin{bmatrix}
4 & -3 & 2 \\
1 & 2 & 7
\end{bmatrix} \begin{bmatrix}
x_{1} \\
x_{2} \\
x_{3}
\end{bmatrix} = \begin{bmatrix}
4x_{1} -3x_{2} +2x_{3} \\
x_{1} + 2x_{2}+7x_{3}
\end{bmatrix}
$$We make a couple of observations:
- (1) This matrix-vector product looks suspiciously like a system of linear equation, if $\vec x$ is a column vector of variables.
- (2) We can write out $A\vec x$ further as a sum of things: $$
x_{1}\begin{bmatrix}
4 \\
1
\end{bmatrix}+
x_{2} \begin{bmatrix}
-3 \\
2
\end{bmatrix}+
x_{3} \begin{bmatrix}
2 \\
7
\end{bmatrix},
$$which is just a **linear combination** of the columns of $A$, with coefficients $x_{1}, x_{2},x_{3}$!
How curious! Observation (1) gives a way to reformulate system of linear equations as a matrix-vector equation, let us keep this in mind for now, we return to this in a bit. For (2), this gives a new way to multiply matrices:
> **Matrix-vector product.**
> Suppose $$A=\begin{bmatrix}| & | & & | \\ \vec a_{1} & \vec a_{2} & \cdots & \vec a_{{k}} \\ | & \ | & & |\end{bmatrix}\quad \text{and }\quad \vec x = \begin{bmatrix}x_{1} \\ x_{2} \\ \vdots\\ x_{k}\end{bmatrix}$$ where $A$ is an $n\times k$ matrix, whose $i$-th column is $\vec a_{i}$, and each column is of size $n\times 1$, and if $\vec x$ is a $k\times 1$ column vector. Then the matrix-vector product $A\vec x$ is just a **linear combination** of the columns of $A$, with corresponding coefficients in the entries of $\vec x$: $$
A\vec x=x_{1} \vec a_{1} + x_{2} \vec a_{2}+\cdots + x_{k}\vec a_{k}.
$$
This is neat and provides an alternate way of computing matrix-vector products. Let us see an example first, then we prove it.
**Example.** Consider the following matrix-vector computation using this rule: $$
\begin{align*}
\begin{bmatrix}
3 & 9 & 7 & 5 \\
2 & -1 & 3 & 4 \\
4 & 2 & -1 & 1
\end{bmatrix} \begin{bmatrix}
\colorbox{lightgreen}4 \\
\colorbox{cyan} {\(-2\)} \\
\colorbox{pink} 8 \\
\colorbox{beige} 2
\end{bmatrix} &= \underbrace{\colorbox{lightgreen} 4 \begin{bmatrix}
3 \\
2 \\
4
\end{bmatrix} \colorbox{cyan} {\(-2\)} \begin{bmatrix}
9 \\
-1 \\
2
\end{bmatrix} +\colorbox{pink} 8\begin{bmatrix}
7 \\
3 \\
-1
\end{bmatrix} + \colorbox{beige}2\begin{bmatrix}
5 \\
4 \\
1
\end{bmatrix}}_{\text{a linear combination of columns of the matrix!}} \\
& = \begin{bmatrix}
12 \\
8 \\
16
\end{bmatrix} + \begin{bmatrix}
-18 \\
2 \\
-4
\end{bmatrix} + \begin{bmatrix}
56 \\
24 \\
-8
\end{bmatrix} + \begin{bmatrix}
10 \\
8 \\
2
\end{bmatrix} \\
&=\begin{bmatrix}
60 \\
42 \\
6
\end{bmatrix}.\quad\blacklozenge
\end{align*}
$$**Remark.** It may seem like this is "multiplication with extra steps", but it shows **structurally** what is happening when you multiply -- that we are merely making **linear combinations** of columns of some matrix.
The proof of this follows from direct computation using the definition of matrix product and careful factoring. It is not super illuminating but straightforward, let me record it here:
$\blacktriangleright$ Proof.
Say we have $$
A = \begin{bmatrix}
a_{11} & a_{12} & \cdots & a_{1k} \\
a_{21} & a_{22} & \cdots & a_{2k} \\
\vdots & & & \vdots\\
a_{n1} & a_{n2} & \cdots & a_{nk}
\end{bmatrix} = \begin{bmatrix}
| & | & & | \\
\vec a_{1} & \vec a_{2} & \cdots & \vec a_{k} \\
| & | & & |
\end{bmatrix} \ \ \text{and}\ \ \vec x = \begin{bmatrix}
x_{1} \\
x_{2} \\
\vdots \\
x_{k}
\end{bmatrix}
$$then, $$
\begin{align*}
A\vec x & = \begin{bmatrix}
a_{11}x_{1} + a_{12} x_{2} +\cdots + a_{1k} x_{k} \\
a_{21}x_{1} + a_{22} x_{2} +\cdots + a_{2k} x_{k} \\
\vdots \\
a_{n1}x_{1} + a_{n2} x_{2} +\cdots + a_{nk} x_{k}
\end{bmatrix} \\
&=x_{1}\begin{bmatrix}
a_{11} \\
a_{21} \\
\vdots \\
a_{n1}
\end{bmatrix} + x_{2}\begin{bmatrix}
a_{12} \\
a_{22} \\
\vdots \\
a_{n2}
\end{bmatrix} + \cdots + x_{k}\begin{bmatrix}
a_{1k} \\
a_{2k} \\
\vdots \\
a_{nk}
\end{bmatrix},
\end{align*}
$$ which shows $A\vec x = x_{1}\vec a_{1} + x_{2} \vec a_{2} + \cdots + x_{k}\vec x_{k}$ as claimed. $\blacksquare$
This idea of **linear combination** keeps showing up, let us define it here:
> **Linear combination.**
> If $v_{1},v_{2},\ldots,v_{k}$ are things that can be scaled by scalars and added together, and that $c_{1},c_{2},\ldots,c_{k}$ are some scalars, then the expression $$
c_{1}v_{1} + c_{2}v_{2} + \cdots + c_{k}v_{k}
$$ is a **linear combination** of $v_{1},v_{2},\ldots,v_{k}$ with coefficients $c_{1},c_{2},\ldots,c_{k}$. Sometimes the coefficients are also referred to as **weights**.
Ok, great, you say, but what about **matrix-matrix product**? Above only shows what happens in a **matrix-vector product**. In fact we can generalize this, and it is an important insight of how matrix multiplication works:
> **The matrix-column rule for matrix product.**
> Let $A$ be some $n\times k$ matrix, and let $B$ be some $k\times p$ matrix, where the columns of $B$ are given by $$
B=\begin{bmatrix}
| & | & & | \\
\vec b_{1} & \vec b_{2} & \cdots & \vec b_{p} \\
| & | & & |
\end{bmatrix}
$$then the product $AB$ is distributing $A$ to each of the columns of $B$, namely $$
AB = \begin{bmatrix}
| & | & & | \\
A\vec b_{1} & A\vec b_{2} & \cdots & A\vec b_{p} \\
| & | & & |
\end{bmatrix}.
$$ (And each column $A\vec b_{i}$ can be further interpreted as linear combinations of columns of $A$)
For example, if $A=\begin{bmatrix} 2 &4\\3 & 2\end{bmatrix}$ and $B = \begin{bmatrix} 3 & 1 & -1 \\ 4 & 2 & 0\end{bmatrix}$, then $$
\begin{align*}
AB &=\begin{bmatrix}
A\begin{bmatrix}
3 \\
4
\end{bmatrix} & A\begin{bmatrix}
1 \\
2
\end{bmatrix} & A\begin{bmatrix}
-1 \\
0
\end{bmatrix}
\end{bmatrix} \\
&=\begin{bmatrix}
22 & 10 & -2\\
17 & 7 & -3
\end{bmatrix}
\end{align*}.
$$
Take a moment to digest these **structural results**, which will give us additional insight of what is happening when we carry out these matrix product operations.